The modern imaging technology employs highly sophisticated optical systems, often consisting of dozens individual optical elements. Over the past decades, imaging optics have become increasingly complex in order to provide the light efficiency for such optical systems (e.g. single-lens reflex (SLR) cameras.) Such complexity is typically required for effective usage of the available aperture ratio, as well as for compensation for undesirable artifacts that inherent to a simple lens element.
The elimination or minimizing of non-linear deviations (i.e. aberrations) from the ideal “thin lens” model is an important part of the optimization of the overall imaging system efficiency. The optical aberrations include such artifacts as geometric distortions, chromatic aberration (wavelength-dependent focal plane), spherical aberration (optical axis distance dependent focal length), and coma (angular dependence on focus).
Since each single optical element with spherical surface(s) suffers from the aforementioned artifacts, the combinations of different lens elements have been used, especially when a high-quality imaging is required (e.g. in photography).
The complex aberration compensated optical systems that possess much better geometric imaging properties, however, suffer from other drawbacks, including drastically increased manufacturing cost, weight, lens flare, reduced reliability, etc.
While it is possible to correct the appearance of the chromatic aberration effects using a digital post-processing in some circumstances, in the most of the real-world circumstances, chromatic aberration results in a permanent loss of some image detail.
The proposed invention uses an alternative approach to achieve the high-quality imaging. Namely, instead of more complex optics, the performance improvement is achieved by adding a pre-process or post-process computational component to correct aberrations of the optical system. Such optical component will be hereinafter referred to as a “digital lens” element of the optical system.
There are various methods of computational aberration correction that have been developed and reported. Thus, the lens-profile-based image correction methods typically take the known characteristics of optical lens/system into account for (automatic) correction of various types of lens distortion, such as color fringes at high contrast edges, vignetting, etc.
Indeed, the detailed knowledge of the optical system used to produce the image could play an important role in correcting of the undesirable artifacts in the image. Due to the complexity of the chromatic aberration (relationship to focal length, etc.), the camera manufacturers employ various lens-specific techniques to minimize chromatic aberration appearance.
Nowadays, almost every major camera manufacturer enables some form of chromatic aberration correction, both in-camera and via their proprietary software. Third party software tools (e.g. PTLens, DxO Optics Pro, Adobe Photoshop Lightroom) are also capable of performing complex chromatic aberration appearance reduction with corresponding databases of cameras/lens.
For example, there is a method reported in U.S. Pat. No. 6,862,373 by Enomoto, describing acquisition of both the input data from an image taken by an optical element (lens) and the information about the very lens been used to record this image. The method further describes image processing using information about the focal length and an aperture (i.e. lens iris opening) at the time of recording, as well as lens characteristics to correct aberrations and vignetting in the image.
Another example, as disclosed in Japanese Patent No. 11-161773 by Habu, also describes correcting magnification chromatic aberration without using any optical components. The magnification chromatic aberration data for the lens for each color is pre-stored, and image processing performs enlarging and reducing the image based on the mentioned pre-stored data, thus performing the magnification aberration correction every time an image is obtained through this lens. Then, after magnification correction, the images of each color are combined into a single image, accomplishing the magnification chromatic aberration correction.
There is another method disclosed in U.S. Pat. No. 7,425,988 by Okada (and, similarly, in U.S. Pat. No. 8,508,655 by Suto) that describes magnification or reduction of a picture on each color; a data memory unit to store the chromatic aberration data specific to the imaging lens for each color (including plurality of zoom, focus and aperture values) and a processing unit that controls the conversion factor and the coordinates magnification aberration correction, using both chromatic aberration data (stored in data memory unit) and the detected image (along with the current zoom, focus and aperture values).
In an ideal situation, the post-processing to remove or correct lateral chromatic aberration would require scaling the fringed color channels, or subtracting some of a scaled versions of the fringed channels, so that all channels spatially overlap each other correctly in the final image (e.g. in holographic microscopy).
In practical applications, however, even a theoretically perfect post-processing-based chromatic aberration reduction system does not increase the image detail in comparison to well-corrected physical lens.
From the chromatic aberration perspective, the reasons for this are following: i) A computational rescaling is only applicable to lateral (not longitudinal) chromatic aberrations. ii) The individual rescaling of color channels results in some resolution loss. iii) Chromatic aberration occurs across the light spectrum, yet most camera sensors only capture a few discrete (e.g. RGB) color channels.
Some chromatic aberration cross-channel color contamination is unavoidable in camera sensors.
Since the above problems are closely related to the content of the particular captured image, no reasonable amount of programming and knowledge of the capturing equipment (e.g., camera and lens data) can overcome such limitations completely.
The disclosed method proposes a new, improved non-blind deconvolution approach for electronic optical aberrations correction. Like the other aforementioned methods, the disclosed method is also based on knowledge (i.e. profiling) of the optical system used for imaging. Furthermore, the method consequently processing the arbitrary captured scene with ‘digital lens’ element of the present disclosure using the profile that is already known for the imaging system.
Compared to other aberration correction techniques, however, the disclosed profiling approach is inherently different, essentially, by utilizing a point-spread function (PSF) extraction for different image scales (i.e. image details) and subsequent artificial neural-network (NN) training. The PSF is an important property in predicting of a light propagation and imaging system performance.
The disclosed method and ‘digital lens’ element expand the applicability of the digital lens from typical image capture systems (digital cameras) towards a broader imaging applications, including augmented reality (AR)/virtual reality (VR) display systems, headsets, viewfinders, etc.
Further features and aspects of the present invention will become apparent from the following description of preferred and optional embodiments with reference to the attached drawings.